Integration of Leaky-Integrate-and-Fire-Neurons in Deep Learning Architectures

Up to now, modern Machine Learning is mainly based on fitting high dimensional functions to enormous data sets, taking advantage of huge hardware resources. We show that biologically inspired neuron models such as the Leaky-Integrate-and-Fire (LIF) neurons provide novel and efficient ways of information encoding. They can be integrated in Machine Learning models, and are a potential target to improve Machine Learning performance. Thus, we derived simple update-rules for the LIF units from the differential equations, which are easy to numerically integrate. We apply a novel approach to train the LIF units supervisedly via backpropagation, by assigning a constant value to the derivative of the neuron activation function exclusively for the backpropagation step. This simple mathematical trick helps to distribute the error between the neurons of the pre-connected layer. We apply our method to the IRIS blossoms image data set and show that the training technique can be used to train LIF neurons on image classification tasks. Furthermore, we show how to integrate our method in the KERAS (tensorflow) framework and efficiently run it on GPUs. To generate a deeper understanding of the mechanisms during training we developed interactive illustrations, which we provide online. With this study we want to contribute to the current efforts to enhance Machine Intelligence by integrating principles from biology

How deep is deep enough? -- Quantifying class separability in the hidden layers of deep neural networks

Deep neural networks typically outperform more traditional machine learning models in their ability to classify complex data, and yet is not clear how the individual hidden layers of a deep network contribute to the overall classification performance. We thus introduce a Generalized Discrimination Value (GDV) that measures, in a non-invasive manner, how well different data classes separate in each given network layer. The GDV can be used for the automatic tuning of hyper-parameters, such as the width profile and the total depth of a network. Moreover, the layer-dependent GDV(L) provides new insights into the data transformations that self-organize during training: In the case of multi-layer perceptrons trained with error backpropagation, we find that classification of highly complex data sets requires a temporal {\em reduction} of class separability, marked by a characteristic 'energy barrier' in the initial part of the GDV(L) curve. Even more surprisingly, for a given data set, the GDV(L) is running through a fixed 'master curve', independently from the total number of network layers. Furthermore, applying the GDV to Deep Belief Networks reveals that also unsupervised training with the Contrastive Divergence method can systematically increase class separability over tens of layers, even though the system does not 'know' the desired class labels. These results indicate that the GDV may become a useful tool to open the black box of deep learning

Sparsity through evolutionary pruning prevents neuronal networks from overfitting

Modern Machine learning techniques take advantage of the exponentially rising calculation power in new generation processor units. Thus, the number of parameters which are trained to resolve complex tasks was highly increased over the last decades. However, still the networks fail - in contrast to our brain - to develop general intelligence in the sense of being able to solve several complex tasks with only one network architecture. This could be the case because the brain is not a randomly initialized neural network, which has to be trained by simply investing a lot of calculation power, but has from birth some fixed hierarchical structure. To make progress in decoding the structural basis of biological neural networks we here chose a bottom-up approach, where we evolutionarily trained small neural networks in performing a maze task. This simple maze task requires dynamical decision making with delayed rewards. We were able to show that during the evolutionary optimization random severance of connections lead to better generalization performance of the networks compared to fully connected networks. We conclude that sparsity is a central property of neural networks and should be considered for modern Machine learning approaches

Predictive coding and stochastic resonance as fundamental principles of auditory phantom perception

Mechanistic insight is achieved only when experiments are employed to test formal or computational models. Furthermore, in analogy to lesion studies, phantom perception may serve as a vehicle to understand the fundamental processing principles underlying healthy auditory perception. With a special focus on tinnitus—as the prime example of auditory phantom perception—we review recent work at the intersection of artificial intelligence, psychology and neuroscience. In particular, we discuss why everyone with tinnitus suffers from (at least hidden) hearing loss, but not everyone with hearing loss suffers from tinnitus. We argue that intrinsic neural noise is generated and amplified along the auditory pathway as a compensatory mechanism to restore normal hearing based on adaptive stochastic resonance. The neural noise increase can then be misinterpreted as auditory input and perceived as tinnitus. This mechanism can be formalized in the Bayesian brain framework, where the percept (posterior) assimilates a prior prediction (brain’s expectations) and likelihood (bottom-up neural signal). A higher mean and lower variance (i.e. enhanced precision) of the likelihood shifts the posterior, evincing a misinterpretation of sensory evidence, which may be further confounded by plastic changes in the brain that underwrite prior predictions. Hence, two fundamental processing principles provide the most explanatory power for the emergence of auditory phantom perceptions: predictive coding as a top-down and adaptive stochastic resonance as a complementary bottom-up mechanism. We conclude that both principles also play a crucial role in healthy auditory perception. Finally, in the context of neuroscience-inspired artificial intelligence, both processing principles may serve to improve contemporary machine learning techniques

Treatment of keratinocytes with 4-phenylbutyrate in epidermolysis bullosa: Lessons for therapies in keratin disorders

Missense mutations in keratin 5 and 14 genes cause the severe skin fragility disorder epidermolysis bullosa simplex (EBS) by collapsing of the keratin cytoskeleton into cytoplasmic protein aggregates. Despite intense efforts, no molecular therapies are available, mostly due to the complex phenotype of EBS, comprising cell fragility, diminished adhesion, skin inflammation and itch.Methods: We extensively characterized KRT5 and KRT14 mutant keratinocytes from patients with severe generalized EBS following exposure to the chemical chaperone 4-phenylbutyrate (4- PBA).Findings: 4-PBA diminished keratin aggregates within EBS cells and ameliorated their inflammatory phenotype. Chemoproteomics of 4-PBA-treated and untreated EBS cells revealed reduced IL1β expression- but also showed activation of Wnt/β-catenin and NF-kB pathways. The abundance of extracellular matrix and cytoskeletal proteins was significantly altered, coinciding with diminished keratinocyte adhesion and migration in a 4-PBA dose-dependent manner.Interpretation: Together, our study reveals a complex interplay of benefits and disadvantages that challenge the use of 4-PBA in skin fragility disorders

Koloniestruktur und kollektive Dynamik der Gattung Aptenodytes bei der Brut

Die Gesetze der Physik beschreiben die unbelebte Welt mit großer Genauigkeit und werden beständig verfeinert, um sie noch detaillierter zu beschreiben und zu verstehen. Die Anwendung dieser physikalischen Gesetze ist jedoch nicht auf die passive Materie beschränkt. Zum Beispiel die Methoden der statistischen Physik, die beschreiben, wie die Eigenschaften der Materie aus der Wechselwirkung von Atomen oder Molekülen entstehen, können auf biologische Systeme von Tiergruppen übertragen werden. In dieser Arbeit werden die Pinguinkolonien der beiden Vertreter der Gattung Aptenodytes untersucht — der Königs- (Aptenodytes patagonicus) und der Kaiserpinguin (Aptenodytes fosteri). Diese Vögel bilden während der Brutzeit Kolonien, die oft aus Tausenden von Individuen bestehen, und sind daher ideal für die Untersuchung von emergenten Phänomenen, die bei der Interaktion zahlreicher Individuen auftreten. Während die meisten Pinguinarten ortsfeste Nester bauen, was keine Dynamik in Brutkolonien zulässt, tragen die Aptenodytes Pinguine ihr einzelnes Ei auf ihren Füßen, und so bleiben die Kolonien auch während der Brut dynamisch. Daher wurden diese beiden Arten ausgewählt, um zu untersuchen, wie kollektive Dynamik mit physikalischen Gesetzen beschrieben werden kann. Diese Arbeit untersucht das Verhalten beider Pinguinarten anhand von Zeitrafferaufnahmen aus Observatorien in der Antarktis und den subantarktischen Inseln. Im Rahmen dieser Arbeit wurden spezielle Software-Werkzeuge entwickelt, um die aufgenommenen Bilder auszuwerten und die Struktur und Dynamik von Königs- und Kaiserpinguinkolonien zu analysieren. Interessanterweise weisen die Kolonien beider Arten ähnliche Strukturen auf wie man sie in physikalischen Systemen finden kann. Während Brutkolonien von Königspinguinen eine flüssigkeitsähnliche Ordnung aufweisen, die über längere Zeiträume stabil ist, zeigen die Kaiserpinguine beim Huddling kristalline Strukturen, die hochdynamisch sind. Diese Arbeit analysiert die Struktur der Brutkolonien von Königspinguinen anhand der Positionen von Tausenden von Königspinguinen aus Luftbildern. Die Charakterisierung dieser Struktur durch die radiale Verteilungsfunktion zeigt auffallende Ähnlichkeiten mit zweidimensionalen Flüssigkeiten, wobei jeder Brutplatz einem Atom der Flüssigkeit entspricht. Brütenden Pinguine, bei denen der Partner gerade in der Kolonie anwesend ist, besetzen in dieser zweidimensionalen Flüssigkeit nur einen Gitterpunkt, genau wie ein einziges Bruttier. Dennoch zeigt die Analyse, dass zwischen den Brütenden eine 25%ige Variation des “Pick”-Radius, d.h. der Abstoßung zwischen Pinguinen, besteht. Die Stärke der Abstoßung zwischen den Pinguinen hängt folglich stark davon ab, wie aggressiv ein brütendes Tier seinen Platz verteidigt. Die Brutplätze sind während der gesamten Brutzeit sehr stabil, was darauf hindeutet, dass die Flüssigkeitsstruktur in der Brutkolonie in einen glasartigen Zustand gekühlt wurde, was als Kompromiss zwischen hoher Dichte und Flexibilität angesehen werden kann, um auf äußere Störungen, z.B. durch die Kolonie gehende Robben, zu reagieren. Wenn die Küken der Königspinguine geschlüpft und alt genug sind, um längere Zeit allein in der Kolonie zu verbringen, bilden sie auch kollektive Strukturen. Diese Crèches, wie sie genannt werden, sind dichte Gruppen von Küken, im Gegensatz zu den weniger dichten Strukturen brütender Erwachsener. Diese Arbeit zeigt eine mögliche Erklärung dafür auf, wie sich diese Cluster bilden: Wiederkehrende Angriffe von Raubvögeln auf ungeschützte Küken, die vor dem Raubtier fliehen, induzieren Clusterbildung, auch wenn keine attraktiven Wechselwirkungen zwischen den Küken vorhanden sind. Im Gegensatz zu Königspinguinen haben Kaiserpinguine keine Territorialität und können daher dichte, kristalline Strukturen in ihren Huddles bilden. Diese Huddles sind unerlässlich, um Energie zu sparen, wenn Kaiserpinguine während des kalten und stürmischen antarktischen Winters brüten. Die Huddles sind jedoch nicht statisch wie die brütenden Königspinguine, sondern zeigen kleine periodische wellenartige Bewegungen. Die Entstehung und Verbreitung dieser Bewegungen werden in dieser Arbeit anhand eines Modells, das von Beschreibung von Verkehrsstaus abgeleitet wurde, analysiert. Die Analyse zeigt, dass diese kleinen Bewegungen im Laufe der Zeit zu einer großflächigen Translokation, Reorganisation und Verdichtung des Huddle führen und kleinere Huddle zusammenwachsen lässt. Das Modell zeigt in Übereinstimmung mit den Daten, dass jeder Pinguin im Huddle die wellenartigen Bewegungen auslösen kann. Die statistischen Schwankungen der einzelnen Auslöseereignisse solcher Wellen werden durch die große Anzahl von Tieren im Huddle vermindert, was zu regelmäßigeren Intervallen in größeren Huddels führt. Runde, rotierende Huddle-Strukturen, die auch in dieser Arbeit auch detailliert analysiert werden, bieten einen noch besseren Schutz gegen die Umwelt, da alle Pinguine nach innen zeigen und ihre empfindliche Vorderseite vor der Kälte schützen. Das vorgestellte Modell sagt voraus, dass die Geschwindigkeit der Pinguine linear mit dem Abstand von der Mitte des Huddle zunimmt, was zeigt, dass sich die Huddles wie starre Platten drehen. Die Winkelgeschwindigkeit der Huddles ist indirekt proportional zum Durchmesser der Huddles, was durch die Verfolgung von Pinguinen am Rand der Huddle in Zeitrafferaufnahmen bestätigt wird. Diese Ergebnisse vertiefen unser Verständnis wie sich diese einzigartigen Vögel in lebensfeindlichen Umgebungen fortpflanzen können. Die neuen Modellbeschreibungen können helfen vorherzusagen, wie die Kolonien auf Veränderungen dieser fragilen Ökosysteme durch globalen Wandel oder menschliche Interaktion reagieren.The laws of physics describe the inanimate world with great accuracy and are constantly refined to describe and understand it in even more detail. The application of these physical laws is not limited to passive matter. For example, methods in statistical physics to describe how the properties of matter emerge from the interaction of atoms or molecules can be transferred to biological systems of groups of animals. In this thesis, the penguin colonies of the two representatives of the Aptenodytes genus, the king (Aptenodytes patagonicus) and the emperor penguin (Aptenodytes fosteri), are investigated. These birds form colonies during breeding that often consist of thousands of individuals, and are therefore ideal for the investigation of emergent phenomena that occur when numerous individuals interact. While most penguin species build static nests, which impedes any dynamics in breeding colonies, the Aptenodytes penguins carry their single egg on their feet, and thus the colonies remain dynamic even during breeding. Therefore, these two species were selected to investigate how collective dynamics can be described with physical laws. In this thesis, the behaviour of both penguin species is analysed using time-lapse images from observatories in Antarctica and the subantarctic islands. Special software tools have been developed in the course of this thesis, to evaluate the recorded images, and to analyse the structure and dynamics of king and emperor penguin colonies. Interestingly, the colonies of both species display structures similar to those found in physical systems. While colonies of breeding king penguins show liquid-like order that is stable over prolonged periods of time, emperor penguins show crystalline structures during huddling that are highly dynamic. This thesis analyses the colony structure of breeding king penguins using positions of thousands of king penguins mapped from aerial images. The characterization of this structure by the radial distribution function shows striking similarities to two-dimensional liquids, with each breeding site corresponding to one atom of the liquid. Breeders where the partner is present in the colony, due to shift changing during breeding, occupy only one lattice spot in this two-dimensional liquid, just like a single breeder. Nevertheless, the analysis reveals a 25% variation of the “pecking”-radius between the breeders, i.e. the repulsion between penguins. This shows that the strength of the repulsion between the penguins strongly depends on how vigorously a breeder defends its place. The breeding sites are very stable throughout the breeding season, indicating that the liquid structure in the breeding colony has been quenched to a glassy state, which serves as a compromise between high density and flexibility to react to external disturbances, e.g. elephant seals passing through the colony. When the king penguin chicks have hatched and are old enough to spend prolonged time on their own in the colony, they also form collective structures. These crèches, as they are called, are dense groups of chicks, unlike the less dense structures of breeding adults. The thesis presents a possible explanation for how these clusters are formed: Recurrent attacks by birds of prey on unprotected chicks fleeing from the predator induce clustering, even in the absence of an attractive potential between the chicks. In contrast to king penguins, emperor penguins lack any territoriality and can therefore form dense, crystalline structures in their huddles. These huddles are essential to save energy when emperor penguins breed during the cold and stormy Antarctic winter. These huddles are not static like king penguin breeding colonies, but show small periodic wave-like movement. The origin and spreading of these movements are analysed in this thesis using a model derived from traffic jam descriptions. The analysis shows that this small-scale movement leads over time to large-scale translocation, reorganisation and compaction of the huddle, and allows smaller huddles to merge. The model shows, in accordance with the data, that every penguin in the huddle can trigger the wave-like movement. The statistical fluctuations of individual triggering events of such waves are diminished by the large number of animals in the huddle, leading to more regular intervals in larger huddles. Round, rotating huddle structures, which are also analysed in detail in this thesis, provide even better protection against the elements, since all penguins face inwards, protecting their sensitive front from the cold. The presented model predicts that the velocity of the penguins increases linearly with the distance from the centre of the huddle, showing that huddles rotate like rigid plates. The angular velocity of the huddle is indirectly proportional to the diameter of the huddle, which is confirmed by tracking penguins at the border of huddles in time-lapse images. These results deepen our understanding of how king and emperor penguins can survive in harsh environments. The models developed in this thesis can help to predict how the colonies can react to changes that occur when their fragile ecosystem is disturbed by global change or human interaction

Physics in Penguin Colonies

In polar regions, highly adapted social behavior is crucial for the survival of several species. One prominent example is the huddling behavior of Emperor penguins. To understand how Emperor penguins solve the physical problem of movement in densely packed huddles, we observed an Emperor penguin colony (Atka Bay) with time-lapse imaging and tracked the positions of more than 1400 huddling penguins. The trajectories revealed that Emperor penguins move collectively in a highly coordinated manner to ensure mobility while at the same time keeping the huddle tightly packed. Every 30 - 60 seconds, all penguins make small steps, which travel as a wave through the entire huddle. Over time, these small movements lead to large-scale reorganization of the huddle. Our data show that the dynamics of penguin huddling is governed by intermittency and approach to kinetic arrest in striking analogy with inert non-equilibrium systems. We will also present observations from a different Emperor penguin colony (Adélie Land), an Adélie penguin colony (Adélie Land), and a King penguin colony (Crozet Island)